Bridging weak supervision and privacy aware learning via sufficient statistics

We present a first attempt in connecting two areas of statistical learning that have not shared much common ground: weakly supervised learning and privacy aware learning. In the former, we aim to learn models of labeled data, when full information of the labels is not available; the latter concerns the design of algorithms with privacy guarantees for the protection of the data, while trading off utility for learning. We focus on classification with linear separators. There exists a sufficient statistic that summarizes all information from the label variable, the mean operator. The fact is known for a broad set of loss functions. Learning algorithms have exploited this property and overcome the lack of label knowledge, learning with label proportions only. We extend the result with almost no structural assumptions on loss functions and regularizers, and show how the approach is potentially viable for any weakly supervised task. Further, we consider the label as the only sensitive variable to protect, while the rest of the data is of public domain. In this scenario, we propose a simple method based on the Laplacian mechanism that obfuscates the mean operator and feed it to a learning algorithm which (a) enjoys α-label differential privacy, (b) is characterized by a generalization bound under almost no structural assumptions and (c) can be integrated into a secure data-sharing protocol for learning. Remarkably, some known results are recovered with simplified proofs.